0.1 Outline

A lower bound on sample complexity through VC-dimension. Cover's coin problem. 10.2 A lower bound on the sample complexity through VC-dimension In previous lectures it has been shown that the VC-dimension of a concept class gives an upper bound on the number of samples needed to PAC-learn concepts from the class. In particular it has been shown that for VC-dimension d it is possible to PAC-learn with parameters and given that the number of samples m is at least m c 0 d log 1 + 1 log 1 ! (10.1) where c 0 is a xed constant. So, as long as VC-dim(C) is nite, it is possible to PAC-learn concepts from C even though jCj might be innnite. One might ask if there are concept classes of innnite VC-dimension that can be learned from nite sample size, or if at least the upper bound of (10.1) can be improved in any substantial way to, say, O(d log(1==)). The answer to both these questions is \No." It is pretty clear that one cannot PAC-learn a concept class C of innnite VC-dimension. Say that we have an algorithm that makes m queries to the examples oracle (m depends only on and), and let Y be a subset of the domain of size 2m that is shattered by C. After m examples A knows nothing about at least half the points in Y , and therefore it cannot predict the labels on those points better than a random coin toss. In fact, for nite VC-dimension equation (10.1) is nearly tight, and we will show that for VC-dimension d, achieving error rate requires c 1 d== samples for some xed constant c 1 .